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Abstract.  Soldiers  incurred  injuries  or  even  lost  their  lives  due  to  rollovers  while  driving  military 
vehicles.  A  recent  report  identified  that  the  one  cause  of  rollovers  is  the  driver’s  inability  to 
assess  rollover  threats,  such  as  a  cliff,  soft  ground,  water,  or  a  culvert  on  the  passenger  side  of 
the  vehicle,  due  to  the  vehicle’s  width.  To  reduce  the  number  of  rollover  accidents,  a  road-edge 
detection  and  driver  warning  system  is  being  developed  to  detect  the  rollover  threats  on  the 
passenger  side  of  the  vehicle  and  warn  the  driver.  This  system  utilizes  a  unique,  ultra-fast 
image  processing  algorithm  based  on  the  neurobiology  of  insect  vision  and  the  study  of  fly 
vision.  The  system  consists  of  a  camera  system,  a  long-range,  planar  laser  scanner,  a 
processing  module  in  which  a  biomimetic  image  processor  detects  edges  present  in  the  images 
in  real-time,  and  a  Driver  Vision  Enhancer  (DVE)  which  displays  the  current  road  image, 
detected  boundaries,  and  road  side  terrain  steepness. 
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1  Introduction 

U.S.  Army  personnel  and  Soldiers  operate  their  military  vehicles  on  rugged  off-road  terrain  and 
in  bad  weather  conditions  while  driving  their  vehicles.  The  Soldiers  often  navigate  and  traverse 
over  rough  terrain  that  could  inadvertently  damage  the  vehicle  and  cause  injury  to  the  crew  and 
driver.  One  of  the  situations  that  the  driver  needs  to  be  concerned  about  is  rollover,  since 
Soldiers  drive  large  military  vehicles  such  as  the  High-Mobility  Multipurpose  Wheeled  Vehicle 
(HMMWV),  Stryker,  Abrams,  Bradley,  or  the  Family  of  Medium  Tactical  Vehicles  (FMTV).  These 
vehicles  are  wide  and  have  hidden  blind  spots  that  hinder  the  driver’s  field  of  view.  Due  to  the 
vehicle’s  width,  the  driver  is  unable  to  assess  rollover  threats,  such  as  driving  close  to  the  edge 
of  the  road  that  has  a  drop-off  into  a  ditch  or  water,  or  a  large  obstacle  that  could  unbalance  the 
vehicle  on  the  passenger  side  of  the  vehicle.  The  driver  can  not  see  these  hazards,  thus  he  or 
she  might  go  over  the  edge,  therefore  causing  the  vehicle  to  rollover.  Such  rollover  incident 
could  be  avoided  if  the  Soldiers  were  equipped  with  technology  to  warn  the  driver  of  potential 
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rollover  threats.  The  Army  is  researching  and  developing  methods  to  decrease  military  accident 
fatalities.  One  technique  that  engineers  and  scientists  are  currently  developing  is  a  road-edge 
detection  and  driving  warning  system  to  notify  the  driver  of  potential  rollover  threats  and  provide 
the  driver  enough  time  to  steer  the  vehicle  away  from  the  threat ,  thus  preventing  a  rollover. 

2  Main  components  of  the  system 

The  Army  is  developing  techniques  to  mitigate  rollover  incidents.  One  technology  that  is  being 
considered,  shown  in  Figure  1,  is  a  system  that  is  being  developed  to  decrease  rollover 
incidents.  The  system  consists  of  a  monochromatic  video  camera,  a  long-wavelength  infrared 
(LWIR)  camera,  and  a  laser  scanner.  This  system  can  be  mounted  on  a  HMMWV  or  any  military 
vehicle  that  is  bulky  and  has  a  limited  field-of-view  out  of  the  passenger  side  of  the  vehicle. 


LWIR  Camera 
(nighttime  view,  smoke, 
fog,  snow,  rain) 

Planar  Laser  Scanner 

up  lo  SO  m  Selection  range  \ 
up  to  90*  scanning  angle  I 
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Video  Camera 
(dear  daytime  view) 
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Figure  1.  Video  and  LWIR  cameras  potential  mounting  positions. 


The  general  flow-chart  of  the  architecture,  which  includes  the  hardware  and  software  for  the 
rollover  detection  system,  is  shown  in  Figure  2.  The  hardware  includes  the  video  and  LWIR 
cameras,  the  laser  scanner,  and  the  Driver’s  Vision  Enhancer  (DVE).  The  video  and  LWIR 
cameras  enable  the  driver  to  see  images  during  daytime  and  nighttime  driving,  and  also  in  poor 
weather  conditions  such  as  fog,  snow,  and  rain.  The  software  reads  in  the  video  images  from 
the  video  or  LWIR  camera  and  processes  the  images.  The  processing  unit  includes  the  fly  eye- 
based  edge  detection,  the  edges  fusion,  the  road-edge  identification,  and  the  roll-over  threat 
detection  algorithms.  The  fly  vision-based  algorithm  integrates  all  the  relevant  edges  that  are 
present  in  the  video  or  LWIR  camera  and  decomposes  them  into  a  single  set  of  edges  or 
contours.  The  road  edge  identification  algorithm  then  defines  the  edges  that  are  relevant  to  the 
road  boundary.  The  rollover  threat  detection  algorithm  determines  whether  the  terrain  steepness, 
which  is  measured  by  the  laser  scanner,  is  negative  or  positive  on  the  passenger  side  of  the 
road.  The  rollover  threat  detection  algorithm  integrates  the  terrain  steepness  with  the  road 
boundary  information  to  assess  rollover  threat.  This  assessment  will  notify  the  driver  visually 
and  acoustically  when  the  system  determines  that  the  driver  is  driving  close  to  the  edge  of  the 
road  and  risks  rolling  the  vehicle. 
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H-2594 


Figure  2.  Flow-chart  of  the  road  edge  detection  system 


2.  1  Hardware:  Cameras  and  Laser  Scanner 

Video  images  have  been  used  to  detect  and  track  road  boundaries  and  lane  markers.  The 
images  obtained  from  the  video  provide  the  best  results  on  roads  with  lane  markers,  during 
daylight,  and  in  good  weather  conditions  (Wang  et  al.,  2004).  However,  video  images  are 
difficult  to  use  during  nighttime,  rain,  snow,  fog,  smoke,  and  on  unmarked  terrain.  To  improve 
viewing,  the  scenery  during  nighttime,  or  in  the  presence  of  smoke,  fog,  snow,  and  rain,  and  on 
all  terrain  conditions,  both  video  and  LWIR  cameras  are  needed. 

LWIR  cameras  allow  the  users  to  capture  images  through  smoke,  fog,  snow  and  rain.  There 
are  two  types  of  infrared  imagery  technologies:  cooled  and  uncooled  infrared  detectors.  The 
Army  is  interested  in  the  uncooled  infrared  detector,  since  it  is  compact  and  affordable.  The 
uncooled  infrared  detector  uses  a  micro-bolometer  which  is  insensitive  to  light,  since  it  can 
operate  in  long-wavelength  infrared  region,  7-14  pm,  thus  making  it  beneficial  for  nighttime 
usage. 

The  images  displayed  from  a  LWIR  camera  allow  the  driver  to  distinguish  the  different  types  of 
materials  used  on  the  roads,  such  as  asphalt,  dirt,  or  gravel,  and  from  the  embankment,  such  as 
grass,  rock,  or  a  cliff.  This  distinction  provides  better  contrast,  thus  making  detection  of  the  road 
boundary  and  embankment  easier  during  nighttime  driving  and  under  different  weather 
conditions.  However,  a  video  camera  and  LWIR  camera  can  not  provide  sufficient  depth 
perception,  since  the  users  can  not  extract  depth  from  the  video  or  LWIR  camera.  It  is  possible 
to  determine  depth  perception  when  multiple  cameras  are  mounted  at  different  angels,  but 
having  multiple  cameras  on  a  vehicle  takes  up  space,  which  is  limited  on  military  vehicles, 
increases  the  overall  cost  of  the  system,  and  increases  the  computational  power  requirements 
such  as  processing  and  memory.  A  laser  scanner  is  needed  to  overcome  this  hurdle. 

The  laser  scanner  provides  depth  or  steepness  perception  since  the  laser  scanner  can  scan  any 
point  in  space  to  indicate  the  presence  of  a  cliff  or  an  obstacle.  An  increase  in  the  slope  of  the 
scanning  denotes  a  terrain  depression  such  as  a  cliff  that  is  on  the  passenger  side,  while  a 
decrease  in  the  slope  will  denote  positive  terrain  such  as  a  big  rock  or  an  obstacle  that  is  on  the 
passenger  side  of  the  vehicle.  Figure  3a  shows  a  parking  lot,  from  left  to  right,  the  flat  surface 
(the  parking  lot),  a  positive  terrain  (grass),  and  a  negative  terrain  (a  drop-off  onto  the  adjacent 
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road).  Figure  3b  shows  the  results  from  the  laser  scanner  algorithm.  The  dark  line  on  the  left 
represents  the  flat  surface,  the  middle  curve  (the  light  color)  represents  the  positive  terrain,  and 
the  slight  curve  on  the  right  represents  the  negative  terrain.  The  threshold  for  positive  and 
negative  terrain  was  set  arbitrarily  to  one  foot,  relative  to  the  flat  surface.  If  the  algorithm  for  the 
scanner  detects  a  negative  or  positive  slope  that  is  greater  than  one  foot,  then  the  slope 
appears  on  the  graph  as  a  positive  or  negative  terrain. 


Positive  Terrain 
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(a) 

Figure  3.  (a) 
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scanner  (b)  Results  from  the  laser  scanner 


The  laser  scanner  that  the  researchers  and  engineers  are  considering  has  a  light  beam  that  is 
invisible  to  the  naked  eye  and  will  not  interfere  with  the  LWIR  imaging.  The  laser  scanner  needs 
to  have  at  least  a  75  meter  range  since  the  scanner  needs  to  measure  the  steepness  in  front  of 
the  vehicle  and  provide  enough  time  to  warn  the  driver  of  potential  barriers  or  drop-offs  and 
enable  the  driver  enough  time,  about  2  seconds,  to  react  and  veer  the  vehicle  away  from  the 
potential  rollover  hazard. 

However,  the  monochromatic  video  and  LWIR  cameras  and  laser  scanner  by  themselves  are 
not  advantageous  to  the  driver,  since  the  cameras  and  scanner  cannot  warn  the  driver  of 
potential  rollover  threats.  Algorithms  are  needed  to  extract  the  images  and  terrain  steepness, 
and  that  information  is  used  to  warn  the  driver  visually  and  acoustically  using  alarm  signals. 

3  Processing  Unit 

After  the  monochromatic  and  LWIR  cameras  capture  the  images  and  the  laser  scanner  returns 
the  measured  distance,  algorithms  are  needed  to  process  the  images  and  analyze  the  distance. 
There  are  four  algorithms  used  1)  a  fly  eye-based  algorithm  to  detect  road  edges,  2)  an 
algorithm  to  fuse  monochromatic  video  and  LWIR  images,  3)  a  road  boundary  identification 
algorithm,  and  4)  a  roll-over  threat  detection  algorithm. 

3. 1  Fly  Eye-based  Road  Edge  Detection  Algorithm 

Viewing  images  from  a  monochromatic  camera  on  a  computer  system  demands  intensive 
computational  power  and  time,  which  is  not  very  ideal,  since  excessive  computational  time 
means  that  there  will  be  a  lag  in  the  images  being  displayed  to  the  driver  on  the  monitor.  This 
lag  in  the  images  can  be  dangerous  to  the  driver  since  the  images  are  not  processed  in  real¬ 
time,  and  the  driver  is  unable  to  determine  whether  the  vehicle  is  close  to  the  edge  of  the  road 
or  not.  To  resolve  the  issue  of  delayed  images  being  displayed  on  the  monitor,  an  innovative 
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technique  to  quickly  process  the  video  images  is  being  implemented  for  the  road-edge  detection 
and  warning  system.  The  technique  that  is  being  utilized  is  based  on  the  cognitive  neurobiology 
of  a  fly.  Since  the  brain  of  a  fly  is  small  and  a  fly  needs  to  maneuver  quickly  to  avoid  obstacles, 
the  fly  flies  by  detecting  the  edges  of  objects  to  avoid  slamming  into  them  (Agassounon  et.  al, 
2004).  Since  the  fly  has  limited  brain  space,  it  needs  to  quickly  see  the  images  and  efficiently 
process  the  images.  This  neurological  process  of  the  fly  has  been  modeled  and  simulated  for 
use  in  detecting  the  boundaries  of  objects  or  roads  . 


Figure  4.  (a)  Image  of  the  road  (b)  Image  using  fly-eye  based  detection 

algorithm 

An  example  of  using  the  fly-eye-based  edge  detection  algorithm  is  shown  in  Figures  4a  and  4b. 
Figure  4a  shows  an  image  of  the  road  and  Figure  4b  shows  the  same  images  using  the  fly-eye- 
based  detection  algorithm.  From  looking  at  Figure  4b,  one  notices  that  the  algorithm  captures 
the  relevant  contour  pixels  of  the  image,  which  are  the  lane  maker,  cliff  edge  and  road  boundary. 

Since  the  fly  registers  the  edges  of  objects,  software  leveraging  this  type  of  cognitive  vision 
responses  augments  the  robustness  and  computational  speed  of  the  system.  The  fly-eye-based, 
edge-detection  algorithm  is  thousands  of  times  faster  than  the  traditional,  gradient-based 
algorithm  because  only  the  relevant  pixels,  such  as  the  boundaries  of  the  objects  in  the  video, 
are  stored,  thus  increasing  the  processing  speed  and  consuming  less  memory  (Sinakevitch  and 
Strausfeld,  2004). 

However,  using  the  ultra-fast,  fly-eye-based,  edge-detection  algorithm  by  itself  is  not 
advantageous  since  images  are  being  captured  by  both  the  LWIR  and  monochromatic  video 
cameras.  The  road-edge  detection  system  does  not  know  which  images  to  use,  thus  to  improve 
the  probability  of  detection,  the  two  images  need  to  be  combined  into  a  single  robust  image  that 
the  system  can  utilize  for  road-edge  detection. 

3.2  Fusion  Algorithm 

When  using  images  from  both  the  video  and  LWIR  cameras,  the  images  do  not  lay  on  top  of 
one  another.  To  overcome  this  minor  stumbling  block,  the  video  and  LWIR  cameras  need  to  be 
mounted  and  oriented  so  that  the  fields-of-view  for  both  cameras  line  up,  thus  enabling  images 
taken  from  both  cameras  to  superimpose  on  top  of  each  other.  The  fusion  algorithm  combines 
the  edges  detected  from  each  of  the  cameras  into  a  single  set  of  edges  that  can  be  displayed  to 
the  driver,  otherwise  the  driver  might  get  dizzy  looking  at  two  different  images. 
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Figure  5.  Edges  fusion  example.  (Images  are  courtesy  of  Zeiss  Optronics.) 

The  fusion  algorithm  uses  a  weighted  sum  of  the  pixel  intensities  of  detected  edges  in  the 
visible  and  LWIR  images.  Thus,  this  method  integrates  the  edge  information  into  a  single-edge 
image.  The  contribution  from  either  the  LWIR  camera  or  the  video  camera  can  be  increased  or 
decreased  by  varying  pixel  intensities.  Figure  5  shows  an  example  of  the  edge  fusion.  The 
picture  on  the  left  is  of  the  road  taken  at  night  using  a  monochromatic  video  camera,  while  the 
picture  on  the  right  is  taken  with  a  LWIR  camera.  The  bottom  picture  is  a  superimposed  image 
of  the  two  pictures  using  the  fusion  algorithm. 

3.3  Road  Boundary  Identification  Algorithm 

In  addition  to  the  fusion  algorithm,  a  road  boundary  identification  algorithm  is  required  since  all 
the  edges  in  the  image  such  as  trees,  big  boulders,  or  buildings  are  displayed.  To  minimize 
confusion  to  the  driver,  the  road  edges  need  to  be  well  defined  and  outlined.  The  road  boundary 
algorithm  utilizes  a  technique  that  Wilson  discovered  called  position  of  pivot  point  estimating 
trajectory  (POPPET)  (Wilson  and  Dickson,  1999).  This  algorithm  capitalizes  on  a  model-based 
technique  to  fit  consecutive  segments  of  the  road  boundary  with  a  curve  fragment  by  fitting  the 
road  edge  segment  with  a  short  2D  vector  using  pivot  points.  This  algorithm  automatically 
searches  for  the  origin  of  the  road  boundary  in  the  lower  right  quadrant  of  the  edge  image  by 
looking  for  the  origin  of  the  bottommost,  rightmost  straight  line.  Then  a  short  vector  of  a  selected 
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length  or  fitting  window  is  drawn  from  the  origin.  The  algorithm  then  counts  the  number  of 
detected  edges  belonging  to  the  vector  by  rotating  the  vector  about  the  origin  point  within  a 
region  defined  by  the  original  reference  to  find  the  best  fit.  The  best  fit  corresponds  to  the 
position  for  which  the  fitting  vector  has  the  most  edge  points  and  represents  the  detected 
fragment  for  the  road  boundary.  The  tip  of  the  vector  becomes  the  new  origin.  The  procedure  is 
repeated  to  detect  the  full  road  boundary  fragment  by  fragment.  Figure  6  shows  an  example  of 
the  road  boundary  algorithm  being  implemented,  where  the  dark  line  in  Figure  6a  and  white  line 
in  Figure  6b  highlight  the  edge  of  the  road. 


Figure  6.  (a)  Black  line  represents  the  road  boundary  (b)  White  line  represents  the  road 

boundary. 


3.4  Rollover  Threat  Detection  Algorithm 

After  the  edges  of  the  road  boundary  have  been  identified,  the  laser  or  radar  scanner  measures 
and  detects  the  presence  of  a  drop-off  or  an  obstacle  on  the  roadside,  and  whether  the  vehicle 
is  about  to  cross  a  certain  threshold  that  would  cause  the  vehicle  to  rollover  if  the  driver  does 
not  steer  back  onto  the  road.  To  calculate  the  road  edge  crossing,  the  rollover  threat  detection 
algorithm  uses  the  vehicle’s  heading,  current  speed  of  the  vehicle,  and  the  distance  to  the  point 
of  rollover.  The  distance  to  the  edge  crossing  point  is  computed  based  on  the  average  speed  of 
the  vehicle  and  the  position  of  the  road  boundary  detected  in  the  image.  When  the  road  edge 
crossing  is  below  a  certain  threshold,  which  is  related  to  the  driver’s  reaction  time,  an  acoustic 
signal  and  a  visual  signal  will  simultaneously  warn  the  driver  of  the  danger.  The  visual  warning 
and  real-time  video  images  will  be  displayed  on  the  DVE. 

3.5  Driver’s  Vision  Enhancer  (DVE) 

The  images  from  the  LWIR  and  monochromatic  video  cameras  are  processed  by  the  algorithms 
and  displayed  on  the  Driver  Vision  Enhancer  (DVE)  as  shown  in  Figure  7.  The  DVE  is  currently 
installed  in  most  military  vehicles.  The  DVE  will  be  used  to  display  the  real-time  images.  The 
real-time  images  are  fused  images  from  the  LWIR  and  monochromatic  cameras  of  the  terrain 
such  as  the  road  boundaries,  and  the  terrain  steepness  information  on  the  passenger  side. 
Other  features  that  will  be  considered  are  GPS,  compass,  distance  to  the  edge,  and  red-flashing 
button  to  notify  the  driver  of  potential  threats.  Figure  8  shows  an  example  of  a  potential  display 
for  the  driver  that  will  be  integrated  on  the  DVE.  The  three-light  threat  status  will  be  used  to 
alert  the  driver  of  potential  rollover  threats.  . 
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Table  1  shows  the  different  warning  stages  of  the  road  edge  detection  system.  When  the  road 
edge  crossing  point  is  detected  at  a  distance  greater  than  75  meters,  no  audio  alarm  is  sounded 
and  all  three  status  lights  remain  green.  When  the  crossing  point  is  between  60m  and  75m, 
there  are  two  green  lights  and  one  red  light,  and  there  is  a  single  beep  sound.  When  the  threat 
is  between  45m  and  60m,  there  are  two  red  lights  and  one  green  light,  and  there  is  a  double 
beep  sound.  When  the  threat  is  between  25m  and  45m  there  are  three  red  lights  and  a  triple 
beep  sound.  Finally,  when  the  threat  is  within  25m  of  the  vehicle,  there  are  three  red  lights  and 
a  repeating  triple  beep  sound  that  lasts  as  long  as  the  threat  remains  within  the  25m  range.  The 
choice  of  75  m  is  arbitrary,  but  allows  at  least  two  seconds  of  reaction  time  to  the  driver  before 
the  vehicle  can  reach  the  threat.  Two  seconds  represents  a  normal  reaction  time  for  an  average, 
healthy,  sober  driver.  A  vehicle  moving  at  a  speed  of  50  MPH  travels  75  meters  in  about  3.4 
seconds.  Therefore,  this  range  appears  as  a  safe  distance  that  leaves  enough  time  for  the 
driver  to  react  and  maneuver  the  vehicle  appropriately.  The  DVE  and  audio  system  will  be 
further  investigated  when  the  developed  system  is  tested  on  a  vehicle. 


Figure  7.  Driver’s  Vision  Enhancer  (DVE)  uses  to  display  the  images  in  the  vehicle 


Figure  8.  Conceptual  design  of  the  information  shown  to  the  driver 
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Table  1.  Warning  Stages 


Warning 

Distance/Range 

Light 

Sound 

No  Detection 

75m  <  d 

- 

1st  stage 

60m  <  d  <  75m 

G-R 

Single 

2nc*  stage 

40m  <  d  <  60m 

G-R-R 

Double 

3rc*  stage 

25m<  d  <  45m 

R-R-R 

Triple 

stage 

d  <  25m 

R-R-R 

Triple  Repeating 

4  CONCLUSION 

Further  work  is  still  being  done  on  this  system.  The  researchers  and  engineers  are  planning  to 
use  a  radar  scanner  instead  of  a  laser  to  measure  the  terrain  steepness,  since  the  radar 
sensors  are  more  cost  efficient  for  the  Army.  The  Army  is  also  planning  to  implement  and  field 
test  the  system  on  a  HMMWV  and  have  Soldiers  drive  the  vehicle  over  an  off-road  terrain  to 
assess  the  system  performance  and  determine  which  components  of  the  road  edge  detection 
system  need  to  be  optimized  to  maximize  the  system.  Furthermore,  the  Army  is  planning  to  do 
an  evaluation  of  how  the  soldiers  will  react  to  the  road  edge  detection  systems  while  driving 
under  different  scenarios. 

The  road  edge  detection  and  driver  warning  system  has  the  potential  to  minimize  the  number  of 
rollover  incidents  since  the  system  will  notify  the  driver  of  potential  rollover  threats  and  enable 
the  driver  to  react  in  a  timely  manner  to  the  threats.  This  system  is  helpful  to  Soldiers  who  are 
traversing  on  unknown,  off-road  or  rugged  terrain.  In  addition,  the  system  will  enable  the  driver 
to  avoid  going  into  a  river  or  rolling  over  a  cliff  when  driving  along  the  side  of  riverbank,  canal,  or 
cliff,  especially  when  the  driver  is  driving  at  night,  or  navigating  in  poor  weather  conditions  such 
as  in  fog,  snow  or  a  rain  storm.  This  detection  system  utilizes  a  unique,  ultra-fast,  image- 
processing  algorithm  based  on  the  neurobiology  of  insect  vision,  specifically  the  vision  of  a  fly. 
The  system  consists  of  a  Long-Wavelength  Infrared  (LWIR)  camera  and  a  visible  spectrum 
monochromatic  video  camera  system,  a  long-range  laser  scanner,  and  a  processing  module  in 
which  a  biomimetic  image  processor  detects  edges  in  real-time.  In  addition,  a  Driver’s  Vision 
Enhancer  (DVE)  displays  the  video  road  image,  detected  boundaries  and  road-side  terrain 
steepness,  in  real-time,  for  the  driver.  This  system  has  the  potential  to  be  implemented  on  most 
military  vehicles  that  are  operating  on  off-road  terrain  to  aid  the  driver  and  reduce  preventable 
rollover  incidents. 
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